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PREFACE 



The Research on Evaluation Prbgram is a Northwest Regional 
Educational Laboratory project of research^ develbprnent^ testing^ 
and training designed to create new evaluation methodologies for 
use in education. This document is one of a series of papers and 

^?P§^^?_P^^^^^M_§Y_P'^§f ^^'".^^^f f l.^^^^^^'^?.^^^?^ adjunct 
scholars^ and project collaborators — ail members of a cooperative 
network of colleagues working on the development of new 
methodolog ies • 

How have the evaluation units in state departments of educatibh 
changed in the last five years? Have there been changes in their 
staffing patterns and in the nature and amount of their work? 
Answers to these and related questions are provided in this 
report which summarizes the results of two surveys (one in 1978, 
the other in 1983) which investigated the nature of state 
department evaluation operations. The report ends with 
information on the projected needs of these evaluation units over 
the next few years i 

Mick t. Smith, Editor 
Paper aiid Report Series 
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EVALUATieN UNITS IN StftTE DEPARTMENTS 
OP SroCftTION: A FIVE- YEAR UPDATE 



In order to understand arid iinprdve the practice of 
educattonal evaluation, it is essential that we study the setting 
or context within which that evaluation takes place. We cannot 
adequately train or select evaluation staff if we are not 
faSitiar with the environment within which they will work. We 
cannot in4>rove evaluation riiahageineht until we understand the 
organizational context of evaluation units. We cannot develop 
effective evaluation inethbds unless we know the purposes and 
settings in which they will be applied. Increasing the iinpact 
and utilization of evaluation requires knowledge of 
drganizatibnal influences r communication patterns^ and decision 
making procedures. The majority of educatidnal evaluations take 
place within one of two organizational settings: local education 
agencies (LEAs) or school districts, and state education agencies 
(SEAs) or state departments of education. We must understand 
these settings if we are to improve the evaluation that takes 
place within them. This paper focuses oh the SEA evaluation 
setting. 

Until recently^ information on EEA and SEA evaluation 
settings was based primarily oh personal experience or anecdotal 
reports. Increasingly, however, there have been survey studies 
of these settings (cf. tyon et ai., iS78? Caulley and Smith, 
1978), studies of brgariizatibhal influences on evaluation (cf. 
Bank and Williams, 1981; Smith and Caiiliey, 1982), and even 
national descriptive studies (cf. Boruch and Cdrdray, 1980; 
Raizeh and Rossis 1981). This brief research note adds to that 
growing collectidn by repdrtihg results from a 1983 survey of SEA 
evaluation unitSe A study addressing similar questions had been 



coriducted 5 years earlier in 1978 and the 1983 ihfbrinatibn is 
cdtipared t-Sfith that earlier information; In addition to providing 
descriptive information on a few important characteristics of SEA 
evaluation unitS/ this paper also suggests how those 
characteristics may have changed over the past five years. 
Certainiy the success of our efforts at evaluation training, 
management^ methods developmerit* and improvement of practice will 
depend in part on the stability of the settings within which we 
work and our knowledge of them. 

Sjtody Results 

In 191B, a phbhe survey was conducted of the directors of 
evaluation units in 25 state departments of education (Cauiiey 
and Smith, 1978; Cauiiey and Smithy 1980). The thirteen 
western-most states were included in the study as well as a 
random Sample of 12 of the remaining 37 states. The results 
therefore represented a reasonable cross-section of ail 50 states 
but were biased in favor of the "western cbhditioh." Of the 25 
states contacted, 21 (841) had centralized program evaluation 
units; all 21 units participated in the study; 

In 1983/ a mailed questionnaire survey was conducted of all 
50 state departments of education. Of the 50 states, 37 (74%) 
had centralized program evaluation units and 29 of them (78%) 
provided data for the study. A ribh- respondent bias check 
suggested that the states hot responding tended to have lower 
state populations and school enrollments than the responding 
states, i.e;, these results are slightly biased in favor of the 
more populous states. 

Although both studies were originally conducted for purposes 
other than those described here, iihey each cbhtained similar 
questions allowing for a longitudinal study of the SEA evaluation 
setting; A comparison of the results of the two studies 
follows. The purpose of this presentation is to suggest possible 
ways in which the context of evaluation within state departments 
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of education inay have changed in the past five years and to 
provide an up-to-date picture of some aspects of the current SEft 
evaluation environSenti 



In fcoth studies, SEA evaluation units were asked about the 
number of staff {Full Time Equivalents-PTEs) employed in their 
units. These data are summarized here using «cpioratory data 
analysis techniques for ease of pcesehtatibh and ihterpretatibh 
(cfi Tukeyr 1977? McGaw, 1981). The data on number of staff are 
graphed using box and whisker plots in Figure 1. 

The median FTE in 1978 was 7^ but only 3.5 in 1983 (the 
medians are indicated by the vertical hash marks on the lines 
bisecting the boxes in Figure i) . Fifty percent of the units had 
staffs of 4 to 17 in 1978 (i.e.^ fell within the box in 
Figure 1), but 50 percent had only 3 to 6 FTE in 1983. 
Nihety^five percent of the units in 1978 had staffs between 1 and 
24; 95 percent had staffs of 1 to 8 in 1983 (i.e., fell between 
the whiskers in Figure 1)^. 

It appears that SEA evaluation units have undergone a 
dramatic reduction in personnel. Although other evidence 
supports the conclusion that these units have lost staff in 
recent years (Grayr Caulley# and Smithy 1982) , these present data 
must be interpreted cautiously. A close examination of the 
original survey questions reveals that respondents were asked in 
1983 about the number of prof e ss ional staff in their units while 
the 1978 respondents were only aske?1 the vaguer question of how 
many staff they had. It is hot clear whether the 1978 responses 
reflect both support and professional staff or just 
professional; Staff size has decreased, but probably not as 
dramatically as this cbmparisoh implies. Incidentally^ the 
extreme outlier state in 1978 (72 FTE) was still the most extreme 
case in 1983 (37 FTO) . 
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Figure 1 
SEA Evaluation Onit Staffing 
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Internal versus External Conduct of Studies 



An important aspect of state department of education 
evaluation operation is the extent to which the unit employs its 
own inhouse staff to conduct studies or whether it subcontracts 
its work to external consultants and monitors their performance* 

Data bh the number of evaluation studies conducted ihhbuse by 
SEA evaluation units appear below in Figure 2. The data on 
studies contracted to external consultants appear in Figure 3i 

A median of 24.5 evaluations were conducted ihhbuse in 1978^ 
but only 8.5 in 1983. Fifty percent of the states did between 15 
and 40 internal studies in 1978 compared with 4 to 20 internal 
studies in 1983. Similarly, the range harrowed fbr the bther 
states; 95 percent of the states ranged from 1 to 72 inhouse 
studies in 1978, 1 to 21 studies in 1983. SEA evaitaation units 
appear to be doing far fewer ihhbuse studies than five years agb. 

The median number of externally contracted studies, zero, has 
stayed the same in the past five years. Figure 3 does illustrate 
a reduction in range fbr the number of external evaluatibhs 
similar to that of the internally conducted studies, but the 
numbers are small. In general^ SEA evaluation units do not 
subcbntract many evaluatibhs tb external cbhsultahts, and those 
who do so are decreasing the number of such studies. 

Unit Responsib t lities 

In 1978, the SEA evaluatibh units were asked what percent of 
their time was spent in program evaluatibh activities. Twenty 
units responded, they reported an average of 63.5 percent of 
their time bh program evaluation^ with two states spending 10 
percent or less and six states devoting full time tb program 
evaluation. 

A mbre detailed question was asked in 1983. Units were asked 
tb identify their major duties frbm a list bf nine 
responsibilities. A summary of these data appears in table 1. 
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Figure 2 
Evaluations Cbnducted Inhduse 
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Figure 3 

Evaiuatldns Contracted to External Cbhsultahts 
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Th6 units identified an average of 6 responsibilities each^ with 
every responsibility listed being a iiiajbr duty for at least half 
of the tanits. The two most coiranbn responsibilities were 
Consultation/Technical Assistance (92.6%) and Evaluation Studies 
(81;5%j; feleariy, SEft evaluation units have multiple^ diverse 
duties which vary across the states. 



Table 1 

Major Responsibilities of SEA 
Evaiaation Units in 1983 



Number of States Citing this 
As a Major Responsibility 
Responsibility (Percent. ^=11 



Gonsuitat ion/Technical Assistance 25 (92.6) 

Evaluation Studies 22 (81.5) 

Needs Assessment 20 (74. IJ 

Information Provision 20 (74.1) 

Evaluation Monitoring 19 (70.4) 

Planning 18 (66; 7) 

Research Studies 16 (59;3) 

Testing Program 15 (55; 6) 

Policy Analysis 15 (55.6) 



Unit Problem s and Impra vejnents 

In 1978* the evaluatibh units were asked to list their itfost 
prevalent evaluatibh problems. The 21 states fesponding 
Identified their greatest problems as insufficient staff arid/or 
too many evaluations (33%)* lack of funds (29%), lack bf time 
(24%) , and lack of staff training (19%) ; An almost identical 
list resulted when the 1983 respondents were asked to name the 
major problems they anticipated facing in the next 3 to 5 years: 
limited funds (66%), insufficient and/or untrained staff (62%), 
and increased demands for evaluatibh (38%) . Methbdblbgical 
problems were cited next most frequently (e.g., 1978 — difficulty 
implementing experimental designs; ±983— need for more adaptive 
methods, need for computer methods). 
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While some aspects o! evaluations within SEAs may have 
changed over the last several years> one thing remains the 
same—a major problem continues to be inadequate funds and 
insufficient numbers of well-trained staff to rleet the demands 
placed on the units. 

In 1983, each evaluatibh unit was asked to select those 
elements of the total evaluatibh process that it felt iribst heeded 
improvement over the coming 3 to 5 years if the unit was to 
provide the best possible service. The responses to that 
question are summarized in Table 2. Over half of the 27 units 
responding said that Data Base Maintenance and Study Design wduid 
need the greatest attehtioh. Problem Formation was a close third. 



Table 2 

Evaluation Elements Most Needing 
Improvement Over the Next 3 to 5 Years 



Evaluatibh El e m e n t 

Data Base Maintenance 
Study Design 
Problem Pormation 
Data Collect ion 
Information Reporting 
Evaluation Management 
Data Analysis 



Number of States in 1983 
Citing Element as Needing 
Major Improvemeht 
(Percent^ Ng27) 

18 166.7) 

14 (51;9j 

13 {48;1) 

lb (37;b) 

10 (37.0) 

9 (33.3) 

8 (29.6) 



Finally, the 1983 respondents were asked what topics or 
activities should be the focus of state and federal research and 
develbpmeht effbrts to improve state level evaluatibh. The most 
frequently cited topics were methbds, especially methbds related 
to cost analysis {9), staff development (5), utilization (4), and 
political cbhcerhs (4) • 
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Cbhclusibh 



Based on similar studies conducted in 1978 and 1983, this 
paper provides information oh changes in state department of 
education evaluation units over a five-year period and an update 
on current conditions. 

in brief r there appears to have been substantial reductions 
in the size of SEA evaluation unit staffs. While the majority of 
studies continue to be conducted inhouse rather than contracted 
out to external consultantSr far fewer evaluations are being 
performed. Although the variability across units iri terms of 
size of staff and number of evaluations conducted seems to be 
decreasing, the individual units still respond to a diverse set 
of multiple respbnsibilitiesr from consultation to evaluation 
monitoring to policy analysis. These units would like assistance 
in providing better services through improved data base 
maihtehahcer study design, and problem formation. In addition, 
they continue to cope with the perennial problems of insufficient 
funds and lack of trained staff. 
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FOOTNOTE 



^These computations are based on McNeil's (1977) 

recbinmehdatiohs for computing fence (and whiskers) placement; 
The fifty and ninety-five percent ranges discussed here are 
approximations based on assumptions of normally distributed large 
samples and are only used suggestively r acknowledging the 
problems of small sanple sizes* 
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